In the diagnosis of rectal cancer, if the rectal tumor area can be automatically and accurately segmented from Computed Tomography (CT) images, it will help doctors make a more accurate and rapid diagnosis. Aiming at the problem of rectal tumor segmentation, an automatic segmentation method of rectal tumor based on improved U-Net model was proposed. Firstly, the sub coding modules were embedded in the U-Net model encoder of different levels to improve the feature extraction ability of the model. Secondly, by comparing the optimization performances of different optimizers, the most suitable optimizer was determined to train the model. Finally, data augmentation was performed to the training set to make the model more fully trained, so as to improve the segmentation performance. Experimental results show that compared with U-Net, Y-Net and FocusNetAlpha network models, the segmentation region obtained by the improved model is closer to the real tumor region, and the segmentation performance of this model for small objects is more prominent; at the same time, the proposed model is superior to other three models on three evaluation indexes including precision, recall and Dice coefficient, which can effectively segment the rectal tumor area.